The Scalable Coherent Interface (SCI) - IEEE Communications Magazine

There is rapidly increasing demand for very-high-performance networked communication for workstation clusters, distributed databases, multiprocessors, industrial data acquisition and control systems, shared access to distributed data, and so on. Higherbandwidth hardware using the traditional protocols is not sufficient. Even at 100 Mb/s, and certainly at 250 Mb/s, throughput for many applications is so limited by delays due to architecturally induced inefficiencies, such as software overheads (often hundreds of microseconds), that higher bandwidth generally raises cost without improving performance. A new approach to communication is required, one that can eliminate the delay due to software overheads, if we are to reap the full benefit of the far higher bandwidths that modern hardware can provide SCI solves this problem by using the distributed-shared-memory paradigm, typically offering submicrosecond delays and bandwidths currently in the range of 1250 to 8000 Mbis per network node. This article first reviews the general properties that an appropriate system architecture should have, and introduces an architectural model, the Local Area MultiProcessor, distinguished by i t s shared-memory performance and i ts ability to handle LAN-style distances These desired properties are then considered in more detail, and practical design decisions are made, illustrated by the evolution of the ISO/ANSI/IEEE standard Scalable Coherent Interface (SCI) as it addressed these issues. Finally, the current status of the various SCI follow-on and support projects is reported. The Scalable Coherent Interface (SCI) David B. Gustavson and Qiang ti, Santa Clara University he technology described in this article arose from a project whose goal was a standard interconnect that would enable the use of many mass-produced microprocessors for supercomputer-class computation at low cost. The usefulness of this technology for local area network (LAN) communication was an unexpected consequence of the scalability requirements of that project, so the technology is described in language appropriate for high-speed buses or memory systems as well as for LANs. This work is closely related to the IEEE Std 1394 Serial Bus, and to the IEEE P1394.2 Serial Express project, which has just started. This technology is contrasted with Serial Bus and Serial Express later in this article. The supercomputing goal demands communication among multiple processors at the speed of main memory systems, with submicrosecond latency and multigigabyte-per-second bandwidths. There is no time for software-implemented protocols in such a system, so the communication model must be kept extremely simple: load from and store to memory that is in a single flat address space, but physically distributed among the processors. The physical-layer signaling mechanisms also must be kept very simple so that the interface circuits can run at very high speed without becoming expensive. However, this memory-like model is very general, so once the physical links proved capable of LAN distances the distributed-memory model became capable of LAN applications, backward-compatible with traditional protocols as needed during the transition to this new technology, yet fundamentally more efficient, able to utilize large numbers of gigabyteper-second communication links effectively. The use of the distributed memory model for networking is not new [l, 21, but these schemes had neither the very high bandwidths nor the support for caching provided by the Scalable Coherent Interface (SCI) technology. In the following, we return to the original motivation for developing the SCI (local area multiprocessor); its application to networking will be discussed in the fourth section. To meet the ever-increasing demand for computing power and the ever-decreasing expectation for its cost, the use of highly parallel multiprocessor systems is essential. Only a few of the traditional few-processor supercomputer companies have survived, and they are now adding multiprocessor-based product lines. It has been obvious for a long time that the cost effectiveness of the microprocessor is unexcelled, and microprocessor performance continues to grow exponentially. The difficulty has been learning how to solve hard problems by using many inexpensive processors instead of one extremely powerful (and expensive) processor. The Scalable Coherent Interface is an interconnect technology designed to be scalable and cost-effective. The objectives of the interconnection can be summarized as follows: We seek a scalable abstract specification of the interface to communicating components that are approximately at the level of microprocessors (intelligent I/O devices, memory, bridges to other systems, networks, etc.) so that these components can be assembled into more complex and capable multiprocessor systems. We require support for efficient multiprocessing using both the message-passing and distributed-shared-memory models. Because multiple cached copies of data will be present in most systems, we require this interface to include a cache coherence mechanism to keep these copies consistent. HISTORY AND BACKGROUND he Scalable Coherent Interface was developed by a number T of high-performance bus designers and system architects who had come to understand the fundamental limits to bus technology during their work on Fastbus (IEEE 960) and Futurebus+ (IEEE 896.x). These contemporary buses pushed bus signaling technology to its limits, and provided various architectural features that support the use of multiple processors. However, it was recognized that very soon microprocessor speeds would exceed the capability of any bus to support significant multiprocessing, and our efforts of the preceding decade were fated to a short life. A study group was organized in 1987 to look for some way out of this catastrophe. This Superbus Study Group met from November 1987 through July 1988, examining alternative approaches for providing buslike services while avoiding bus limitations. Gradually the form that the solution would require, if indeed a solution were possible, became clear. In July 1988 an official IEEE Working Group was chartered. The group was very fortunate to have several very talented and experienced core members who were available essentially full-time. Initially all the work52 0163-6804l96l$05.00